Technique for simulating film grain on encoded video

ABSTRACT

Simulating film grain in an encoded image occurs by extracting the film grain and then characterizing the film grain for encoding in a video stream to enable the film grain restoration upon decoding. Typically, the film grain is characterized based either on the type of film, or by using a particular model. In practice, the film grain particulars are transmitted as parallel information to the video coded stream, typically as a film grain Supplemental Enhancement Information (SEI) message when using the ITU T H.264 video coding standard.

This application claims the benefit under 35 U.S.C. § 365 of theInternational Application, PCT/US2004/005365, filed Feb. 24, 2004, whichwas published in accordance with PCT Article 21(2) on Nov. 4, 2004 inEnglish and which claims the benefit of U.S. provisional patentapplication No. 60/462,389, filed Apr. 10, 2003.

TECHNICAL FIELD

This invention relates to a technique for simulating the grain ofphotographic film on a sequence of encoded video.

BACKGROUND ART

The photographic film used to make motion pictures comprisessilver-halide crystals dispersed in an emulsion that is deposited inthin layers on a film base. The exposure and development of thesecrystals form the photographic image, which is made at the largest scaleof discrete particles of silver. With color negatives, tiny blobs of dyeexist on the sites where the silver crystals form after chemical removalof the silver following development. These small specs of dye form the‘grain’ in color film. Grain occurs randomly in the resulting imagebecause of the random formation of silver crystals on the originalemulsion. Within a uniformly exposed area, some crystals develop byexposure, others not. Grain varies in size and shape. The faster thefilm (i.e., the greater the sensitivity to light), the larger the clumpsof silver formed and blobs of dye generated, and the more they tend togroup together in random patterns. The grain pattern is typically knownas ‘granularity’.

The naked eye cannot distinguish individual grains, which vary from0.0002 mm to 0.002 mm. Instead, the eye resolves groups of grains, whichthe viewer identifies as film grain. The larger the image resolution,the more likely the viewer will perceive the film grain. While clearlynoticeable in cinema and high-definition images, film grainprogressively loses prominence in Standard Definition Television (SDTV)images and becomes imperceptible in even smaller formats.

Typically, within the domain of video coding, efforts currently exist toimprove the performance of the encoder to encode film grain at highbit-rates. Note that, since film grain becomes noticeable only in largeimage formats such as Standard Definition Television (SDTV), HighDefinition Television (HDTV) and above, the problem of film grainaffects mainly professional encoders.

Present-day studies about film grain have targeted mainly photographicimaging applications (image edition, medical imaging, satellite imagery,astronomy and astrophotography, etc.). There currently exist softwareapplications in the market place (Adobe After Effects, Photoshop, etc.)that provide solutions to this problem. In the domain of video coding,studies have addressed the desirability of removing film grain with thegoal of improving the coding performance at mid and low bit-rates.However, the literature contains no specific strategies to encode filmgrain differently from other high frequency information, such as textureor contours.

In some sense, film grain constitutes the correlated noise inherent inthe physical process of developing of photographic film. The presence offilm grain denotes ‘real-world’ images contrast to ‘computer-generated’material with no film grain. Images without film grain look likehyper-reality, simply too flat and too sharp to be real. For thisreason, film grain remains a desired ‘artifact’ in most images and videomaterial. However, because film grain arises from a random processaffecting high frequencies, the encoding process typically affects filmgrain. Lossy encoders commonly achieve part of their compression gain byavoiding the transmission of the high frequencies in the DCT domain.Such frequencies are typically associated with noise, sharp edges andtexture, but also with the film grain. Even at high bit-rates, filmgrain becomes difficult to preserve, and always at a high compressioncost.

Thus, a need exists to alleviate the cost of encoding film grain withoutdegrading the perceived quality of the displayed image(s).

BRIEF SUMMARY OF THE INVENTION

Briefly, in accordance with present principles, there is provided amethod for simulating film grain in an encoded image. The providedmethod entails film grain extraction and film grain characterization atthe encoder to enable the film grain restoration at the decoder. In apreferred embodiment, the method commences by identifying the film grainthat is present in an incoming image. Thereafter, the film grain ischaracterized based either on the type of film, or using a particularmodel. The incoming image is then encoded and the film graincharacterization information transmitted which allows a decoder decodingthe encoded image to restore the film grain in accordance with theinformation transmitted. The method transmits the film grain particularsas parallel information to the video coded stream, typically as a filmgrain Supplemental Enhancement Information (SEI) message when using theITU-T H.264 video coding standard.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a block diagram of a first embodiment of a system inaccordance with the present principles for simulating film grain inaccordance with the present principles;

FIG. 2 illustrates a block diagram of a second embodiment of a system inaccordance with the present principles for simulating film grain inaccordance with the present principles; and

FIG. 3 illustrates a block diagram of a third embodiment of a system inaccordance with the present principles for simulating film grain inaccordance with the present principles.

DETAILED DESCRIPTION

Different types of motion picture film have different film grainpatterns well known in the art. Thus, identifying the type of film stockused to record the original source material allows identification of thefilm grain of the encoded material. Table 1 lists an individualnumerical identifier for each of a plurality of exemplary film stocks,each having a known film grain pattern. TABLE 1 Identifier Film grainmodel 0 Kodak Vision 200T 5274 1 Kodak Vision 259D 5246 2 Kodak Vision320T 5277 3 Kodak Vision 500T 5279 . . . N Kodak PROFESSIONAL SUPRA 100

Knowing the type of film originally used to record the pictures embodiedin an encoded video assures optimal quality when restoring the filmgrain following image decoding. Additional information concerningdown-sampling factors that apply to the original image also will provehelpful the film grain restoration at the correct scale. Note that TableI is not intended to be all inclusive but merely exemplary. Other filmstocks from other film manufacturers will have separate identifiers.

In the absence of any knowledge of the type of film stock used to recordthe original source, film grain characterization can occur through amodeling process. The modeling process seeks to reduce the amount offilm grain characterization information to be transmitted by providing acompact representation of the film grain pattern and intensity. Such anapproach provides an estimate of the original film grain, which candiffer from the actual film depending on the selected modeling process.When the system that models the film grain at an encoder is enabled toselect among more than one modeling methods to characterize the filmgrain of the incoming images, a decoder should receive at least someinformation identifying the modeling method that was selected. In aparticular embodiment, the modeling process could provide a compactrepresentation of the film grain according to a non-parametric model. Inanother embodiment, the modeling process could consist in aparameterization process according to a pre-defined mathematical model.To illustrate this last embodiment, Table 2 provides an example ofseveral different mathematical models could be used to describe filmgrain. TABLE 2 Identifier Film grain model 0 f(x, y, c) = d* n 1 f(x, y,c) = s(x, y, c) + k*s(x, y, c) + d* n 2 f(x, y, c) = a * f(x − 1, y − 1,c) + b*f(x, y, c − 1) + d*n 3 f(x, y, c) = a * r(x, y, c) − b*s(x, y,c) + d*n . . . N f(x, y, c) = a*[d(x − 1, y, c) + f(x, y − 1, c)] +b*f(x, y, c − 1) + d*n

The use of parametric models requires the transmission of the estimatedset of parameters. The parameters will depend on the type of model asspecified in Table 2, or in the simplest case, will correspond to aunique film grain model known a priori from the type of film asdescribed in Table 1. The parameters of a given film grain model shouldallow adjustment of the size of the film grain, its intensity, itsspatial correlation, its color correlation, etc. As an example, assumethe following formula serves to model the film grain in an image:f(x,y,c)=a*[f(x−1,y,c)+f(x,y−1,c)]+b*f(x,y,c−1)+d*nwhere f(x,y,c) represents the film grain of the pixel at coordinates(x,y) on the color component c, and n represents a Gaussian noise ofmean zero and variance one. According to this model, an encoder shouldtransmit the parameters ‘a’, ‘b’ and ‘d’ to allow a decoder to simulatethe original film grain. Note that the parameters of the model coulddepend on other factors, such as signal intensity, the color component,etc. Hence, transmission of the film grain model parameters actuallyentails transmission of sets of model parameters for each differentcase.

FIG. 1 depicts a block schematic diagram of a first embodiment of asystem 10 in accordance with the present principles for performing filmgrain simulation. The system 10 includes a Film Grain Remover 22 thatserves to remove the film grain from an input video stream 12 to yield afiltered video stream 24 received at a Video Encoder 13. Film grainremoval constitutes a particular case of noise filtering where the noisesignal appears correlated with the image signal. Thus, the Film GrainRemover 22 can take the form of a classical image filter, although sucha filter will not necessarily provide optimal performance. The VideoEncoder 13 encodes the filtered video stream 24 to yield a coded videostream 14 for receipt at a Video Decoder 15 that decodes the codedstream to yield a decoded video stream 16. The Video Encoder 13 and theVideo Decoder 15 utilize the same video coding scheme as are well knownin the art. For example, the video coding scheme could comprise theITU-T H.264 video-coding standard, or another type of block-basedcoding. Encoders and decoders that utilize the MPEG-2 and the ITU-TH.264 standard are well known.

The system 10 also includes a Film Grain Characterizer 23 that receivesthe input video stream 12 and the filtered video 24 stream. From thesevideo streams, the Film Grain Characterizer 23 outputs film graincharacterization information 25. In its simplest implementation, theFilm Grain Characterizer 23 characterizes the grain in the input video12 from a limited set of film grain samples. As an example, film grainsamples can be obtained by subtracting the filtered video 24 from theinput video 12. In another embodiment, the Film Grain Characterizer 23can take the form of a look up table that outputs film graincharacterization information in accordance with the type of film stockoriginally used to record the picture embodied in the input video stream12. Thus, for example, metadata accompanying the input video stream 12could identify the type of film stock in accordance with Table 1. Usingits look-up table, the Film Grain Characterizer 23 will provide theparameters for the identified film, as well as corresponding film grainmodel for that film. In another embodiment, the Film Grain Characterizer23 can make use of a film grain modeling process that characterizes thefilm grain in the input video stream 13 by a set of parameters estimatedaccording to a pre-defined method.

Following the film grain characterization, a Film Grain characterizationinformation Encoder 26 transmits an encoded information streamcontaining the film grain characterization information to a Film Graincharacterization information Decoder 28 in parallel to the encoded videostream 14 transmitted by the Video Encoder 13 to a the Video Decoder 15.Both the Video Encoder 13 and the Film Grain characterizationinformation Encoder 26 use the same encoding scheme. Thus, for example,when the Encoder 26 utilizes the ITU-T H.264 video-coding standard forencoding, the coded film grain characterization information stream 27can take the form of the film grain Supplemental Enhancement Information(SEI) message as defined in the ITU-T H.264 video coding standard.

The Film Grain characterization information Decoder 28 decodes the codedfilm grain characterization information stream 27 to yield a decodedfilm grain characterization information stream 29 for input to a FilmGrain Restoration Processor 30. In its simplest form, the Film GrainRestoration Processor 30 takes the form of a pattern adder that blends afilm grain pattern with the decoded video stream 16 supplied by theVideo Decoder 15 to yield a decoded video stream with simulated filmgrain 31. In another embodiment, the Film Grain Restoration processor 30can comprise a pattern generation processor that creates a film grainpattern according to the film grain characterization information in thedecoded film grain characterization information stream 29.

Notice that the film grain characterization information can varydynamically through a video sequence. Thus, different groups of framescan require the transmission of different film grain characterizationinformation. In this way, the Film Grain Restoration processor 30 canupdate the film grain pattern depending on the transmittedcharacterization information.

FIG. 2 depicts a second embodiment 10′ of a system for simulating filmgrain in accordance with the present principles. The system 10′ sharesmany of the same elements as the system 10 of FIG. 1 and like referencenumbers describe like elements. Indeed, the system 10′ of FIG. 2 differsonly in the absence of the Film Grain characterization informationEncoder 26 and Film Grain characterization information Decoder 28 ofFIG. 1. The system 10′ of FIG. 2 uses the Video Encoder 13 and VideoDecoder 15 to encode and decode respectively the film graincharacterization information 25 output of the Film Grain Characterizer23. The system 10′ of FIG. 2 requires the use of a video coding standardthat supports the transmission film grain characterization informationas parallel enhancement information. Thus, for example, when the VideoEncoder 13 utilizes the ITU-T H.264 video-coding standard for encoding,the coded film grain characterization information stream 27 can take theform of a Supplemental Enhancement Information (SEI) message as definedin the ITU-T H.264 video coding standard.

FIG. 3 depicts a third embodiment 10″ of a system for simulating filmgrain in accordance with the present principles. The system 10″ sharesmany of the same elements as the system 10′ of FIG. 2 and like referencenumbers describe like elements. Indeed, the system 10″ of FIG. 3 differsonly in the absence of the Film Grain Remover 22 of FIG. 2. The system10″ of FIG. 3 uses the reconstructed images available at the VideoEncoder 13 to simulate the result of removing film grain. The system 10″of FIG. 3 affords two advantages as compared to the systems 10 of FIGS.1 and 10′ of FIG. 2. First, the system 10″ of FIG. 3 reduces thecomputational complexity related to film grain removal, and secondly, itadapts the film grain characterization to the amount of film grainsuppressed by the Video Encoder 13. Once the Film Grain Characterizer ofFIG. 3 disposes of both the input video 12 with film grain, and areconstructed video 24 resulting from Video Encoder 13, it canaccomplish the task of characterizing the observed film grain.

In some cases film grain characterization can involve color conversionand/or pixel sample interpolation depending on the original file format.For high quality applications, film grain modeling occurs in the RGBcolor space, which better approximates the layer configuration of thephysical process of film developing. The simplest parametric model canassume the film grain to be a Gaussian noise of zero mean uncorrelatedwith the image signal. In this case, only the transmission of thestandard deviation of the Gaussian function is required. Morecomplicated models can require the transmission of different parametersfor each color component and/or for different gray level sets. Thechoice of a model can be strongly related to the affordable complexityat the decoder side, the number of bits available for encoding the SEImessage and mainly the desired quality on display.

In accordance with the present principles, film grain encoding isperformed by the step of (a) characterizing the film grain at anencoder, (b) transmitting the film grain characterization information inparallel to the coded video stream and (c) blending the simulated filmgrain with the decoded video. As discussed, film grain simulation canrely on a predefined model, which reproduces the film grain of aspecific type of film, or can occur by parameterization using amathematic model. In all cases, film grain restoration occurs prior todisplay. Images with added film grain are never used within the decodingprocess; however, some parallelization could be possible for causalmodels.

The foregoing describes a technique for encoding film grain in a videoimage.

1. A method for simulating film grain, comprising the steps of:receiving an encoded image receiving film grain characterizationinformation indicative of grain in a film on which the encoded image wasoriginally recorded, decoding at least the encoded image; simulating apattern of film grain in accordance with the received film graincharacterization information; and blending the simulated film grainpattern with the decoded image.
 2. The method according to claim 1further comprising the steps of: receiving the encoded image in an ITU-TH.264 video coding format; and receiving the film grain characterizationinformation as a Supplemental Enhancement Information (SEI) Message. 3.The method according to claim 1 wherein the step of receiving the filmgrain characterization information includes the step of receiving anidentifier of which type of film stock was originally used to record theencoded image.
 4. The method according to claim 1 wherein the step ofreceiving the film grain characterization information includes the stepof receiving an identifier of a model that best approximates the filmgrain in the film stock originally used to record the encoded image. 5.The method according to claim 1 wherein the step of receiving the filmgrain characterization information includes the step of receivinginformation indicative of film grain size, intensity, spatialcorrelation, and color correlation.
 6. The method according to claim 1further including the step of separately simulating the pattern of filmgrain for separate groups of frames in the encoded video.
 7. A methodfor simulating film grain, comprising the steps of: encoding an imageoriginally recorded on film; identifying the film grain present in theinput image; and establishing film grain characterization informationfor the film in accordance with a predefined modeling process so thatupon decoding the encoding image, a pattern of film grain can besimulated in accordance with the film grain characterization informationand blended with the decoded image.
 8. The method according to claim 7further comprising the steps of: encoding the image in an ITU-T H.264video coding format; and formatting the film grain characterizationinformation as a Supplemental Enhancement Information (SEI) Message. 9.The method according to claim 7 wherein the step of establishing thefilm grain characterization information includes the step of identifyingwhich type of type of film stock originally recorded the encoded image.10. The method according to claim 7 wherein the step of establishing thefilm grain characterization information includes the step of identifyinga model that best provides an indication of film grain in the filmoriginally recorded the image.
 11. The method according to claim 10wherein step of identifying the model includes choosing among a bestmodel among a plurality of film grain models.
 12. The method accordingto claim 7 wherein the step of establishing the film graincharacterization information includes the step of establishing filmgrain size, intensity, spatial correlation, and color correlation. 13.The method according to claim 7 further including the step of removingfilm grain from the image prior to encoding.
 14. Apparatus forsimulating film grain in an image, comprising of: a decoder forreceiving an encoded image and for receiving film grain characterizationinformation indicative of grain in a film on which the encoded image wasoriginally recorded and for decoding the image; and a film grainrestoration processor for simulating a pattern of film grain inaccordance with the received film grain parameter information; and forblending the simulated film grain pattern to the decoded image.
 15. Theapparatus according to claim 14 wherein the decoder receives the filmgrain characterization information as parallel information to theencoded image.
 16. The apparatus according to claim 14 wherein thedecoder receives the encoded image in an ITU-T H.264 video codingformat; and wherein the decoder receives the film grain characterizationinformation as a Supplemental Enhancement Information (SEI) Message. 17.The apparatus according to claim 14 wherein the film graincharacterization information includes an identifier of which type offilm stock originally recorded the encoded image to provide anindication of film grain.
 18. The apparatus according to claim 14wherein the film grain characterization information includes anidentifier of a model that best provides an indication of film grain inthe film originally recorded the encoded image to provide an indicationof film grain.
 19. The apparatus according to claim 18 wherein the modelidentifier identifies the best model among a plurality of film grainmodels.
 20. The apparatus according to claim 14 wherein the film graincharacterization information includes information indicative of filmgrain size, intensity, spatial correlation, and color correlation. 21.The apparatus according to claim 14 wherein the film grain restorationprocess separately simulates the pattern of film grain for separategroups of frames in the encoded video.